1 research outputs found
Reasoning over Description Logic-based Contexts with Transformers
One way that the current state of the art measures the reasoning ability of
transformer-based models is by evaluating accuracy in downstream tasks like
logical question answering or proof generation over synthetic contexts
expressed in natural language. However, most of the contexts used are in
practice very simple; in most cases, they are generated from short first-order
logic sentences with only a few logical operators and quantifiers. In this
work, we seek to answer the question how well a transformer-based model will
perform reasoning over expressive contexts. For this purpose, we construct a
synthetic natural language question-answering dataset, generated by description
logic knowledge bases. For the generation of the knowledge bases, we use the
expressive language . The resulting dataset contains 384K
examples, and increases in two dimensions: i) reasoning depth, and ii) length
of sentences. We show that the performance of our DeBERTa-based model,
DELTA, is marginally affected when the reasoning depth is increased and it
is not affected at all when the length of the sentences is increasing. We also
evaluate the generalization ability of the model on reasoning depths unseen at
training, both increasing and decreasing, revealing interesting insights into
the model's adaptive generalization abilities